An Evaluation of the IRT Models Through Monte Carlo Simulation

Item response theory (IRT) models are commonly used in the realm of educational and psychological testing. The three commonly used IRT models for responses that are graded as either correct or incorrect are the One-parameter, Two-parameter, and Three parameter IRT models. These models are used to calibrate standardized tests and to assess human latent traits such as student’s overall intelligence, vocabulary prowess, or mathematical ability. The effectiveness of these tests is regularly debated by students, especially those who are required to take these examinations, as their future may depend on how they are assessed by these IRT models. One good example is the Graduate Record Exam (GRE), which are taken by students and used by universities to assist them in the selection of applicants to their graduate program. The main objective of this research was to study the validity of each IRT model in assessing students’ ability. In particular, the effect of the sample size and exam length to the accuracy of the estimates of students’ ability and item characteristics were studied. The results were based on an extensive Monte Carlo simulation study done using the statistical software R. Through these simulation studies, we hope to discover the IRT model that is most effective in calibrating standardized tests and in assessing student’s raw abilities, and the acceptable exam length and sample size to obtain accurate parameter estimates. Preliminary results indicate that the Three-parameter model is problematic as the estimates of the item characteristics (difficulty, discrimination, and guessing parameters) obtained using this model are biased. Our results in evaluating IRT models confirmed some of our conjectures of the preliminary phase of our investigation. The Three-parameter model proved problematic, as correlations between actual and estimated parameters were deficient. However, the One-parameter IRT model and Two-parameter IRT model performed well as correlations between actual and estimated parameters for difficulty and ability parameters were very well and similar from model to model.